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ABSTRACT 

This paper reports on an independent review of two 
examinations designed to assess teacher competencies: the Praxis 
Series and the Content Mastery Examinations for Educators (CMEE). The 
review was ordered by the Montana Department of Public Education in 
order to decide which battery of tests would replace the National 
Teacher Examination (NTE). The paper, which is a point-by-point 
examination of the Prasix Series and the CMEE, covers: (1) a general 
description of each examination; (2) test content; (3) accommodation 
for people with disabilities; (4) deve 1 opment ; (5) technical and 
psychome t r ic aspects (validity, bias, reliability, and i;core 
reporting); (6) strengths and v;eaknesses; and (7) a discussion of 
each test series. The paper concludes with a genera 1 discussion on 
the role of .assessment of knowledge and skills in various 
professions. Findings of the study indicate that the CMEE and the 
Praxis Series are both appropriate assessment batteries; both have 
been developed and studied to assure validity and reliability. 
However, during the study, while technical assistance was readily 
ava liable from the National Computer System staff regarding CMEE, 
getting information on the Praxis Series from Educational Testing 
Service was more cumbersome. The study concludes that assessment of 
knowledge of candidates for teaching certificates must serve the 
state (representing the children) and the teaching profession by 
assisting to certify only qualified candidates. The process should be 
kept simple and efficient for candidates, assessing the areas the 
state and profession deem necessary, and variables such as score 
reporting, cost, and accessibility ought to be considered when making 
decisions on assessment instruments. (ND) 
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Question Posed: 

With the phasing out of the NTE, do either the CMEE or the Praxis Series 
meet the requirements of a competency-based assessment fcr beginning 
teachers in Montana? 

How the Question was answered: 

Reviews were made of each test, comparing the following aspects : general 
description, test content, accommodations for individuals with disabilities, 
description of test development, technical and psychometric aspects, 
stren^hs and weaknesses. 

Introduction 

In the Spring of 1994, The Board of Public Education in the State of Montana 
requested independent reviews of two examinations designed to assess 
teacher competencies. These were the Praxis Series, developed by 
Educational Testing Service, and the Content Mastery Examinations 
for Educators (CMEE), developed by lOX Assessment Associates for 
National Computer Systems (NCS). It was determined that one of these 
battery of exams would replace the current assessment devise, the National 
Teacher Examination (NTE), because it was being phased out by 
Educational Testing Service. The following is a point by point comparison 
of the two examinations. 



Content Mastery Examinations for Educators 



1. General Description 

The Content Mastery Examinations for Educators (CMEE) is a criterion 
referenced test developed to evaluate beginning teachers. It has 24 separate 
tests of content and basic skills. The Basic Skills Test is intended to be used 
by institutions and state departments to assess students of teacher training 
programs as well as beginning teachers. The Basic Skills Test assesses 
writing, reading and mathematics skills. There is also a Pedagogy exam, 
which assesses knowledge of educational concepts and methods. 

The CMEE also has 22 content area exams, including Art Education, 
Biology, Business Education, Chemistry, Early Elementary Education, 
Earth and Space Science, Elementary Education, English, French, Health 
Education, Home Economics, Mathematics, Music, Physical Education, 
Physics, Reading Specialists, Social Studies, Spanish, and Special 
Education. It also includes tests for Administration and Supervision, 
School Library Media Specialist, and Guidance and Counseling. 
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National Computer Systems (NCS) allows states to set dates for the exam, 
and the number of tests it wishes to offer per year. They also allow states to 
determine the amount of time that is allocated to complete the exam. States 
may also contribute to decisions such as how special accommodations can 
be granted to individuals with disabilities and in other special situations. 

A suggested examination configuration (provided by NCS) for a person 
certif^ng for Elementary and Special Education may be: 

* The Basic Skills Test for either entrance to or exit from a Teacher 
Education Program 

* The Elementary Education content area exam to assess specific 
skills and competencies in Elementary Education 

* The Special Education content area exam to assess specific skills and 
competencies in Special Education 

* The Pedagogy exam to assess teaching methods, classroom 
management, human development and other pedagogical content 

* The total cost of this battery would be $200.00. Cost of the exam is $50.00 
per test. 

2. Test Content 

The content is reported to be representative of the knowledge base that is 
necessary to be a successful beginning teacher. Several panels of experts, 
teachers and teacher educators, were .<iurveyed as to the appropriateness of 
content outlines during the development of the instrument. Questions are 
presented in an educational context in order to be relevant to the test taker. 

The CMEE utilizes several techniques to evaluate skills and abilities. These 
include: a writing sample in The Basic Skills Test, a video taped section in 
The Pedagogy Test, wMch shows classroom situations; a listening and 
speaking portion in the French and Spanish tests; and the use of 
calculators in the mathematics, physics, and chemistry tests. 

The video taped section of The Pedagogy Test is an optional portion of the 
exam, which presents several three minute vignettes of classroom 
instruction at several different grade levels and content areas. Candidates 
are given multiple choice questions on topics such as classroom 
management, instructional methods, and the like. 

NCS is willing to work with state departments to develop individualized, 
open ended and/or short answer questions for any of the exams in order to 
meet the specific testing needs of the state. 
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3. Accommodations for People with Disabilities 

Every attempt to remain within legal parameters are made, and 
accommodations are allowed for anyone facing physical or cognitive 
challenges with the proper documentation. Accommodations are routinely 
made by utilizing large print tests, readers or Braille formats for persons 
with visual impairments, interpreters for individuals with hearing 
impairments, extended time, separate testing rooms, and front of the room 
seating. Information regarding accommodations is available to candidates 
in the registration bulletin. 

4. Development 

The CMEE tests began as content specijdc outlines. These outlines were 
developed by examining content areas and what was believed to be 
important Imowledge for beginning teachers. These outlines guided the 
development of questions, as well as the validation studies. 

National Computer Systems (NCS), the developers, utilized experienced 
teachers in the development and determination of appropriateness of the 
content outUnes, bared upon what they considered to be adequate 
performance by beginning teachers. To develop test questions, NCS did the 
following: 

a. Content Identification Triads: Content identification using triads 
consisting of teacher educators experienced teachers and staff from 
lOX Assessment Associates. 

b. Content Determination Panels: Outlines were defined by panels 
consisting of teachers, administrators, and state department 
officials. 

c. Item Generation: Test items were created according to refined 
outlines. 



d. National Content Review: Over 360 educators reviewed items for 
quality. 

e. National Bias Review: African American, Asian American and 
Hispanic American reviewers examined items for unfair or offensive 
content. 

f. Field Test of Items: Tests were administered in teacher education 
programs. 

g. Technical Review of Items: Field test data were reviewed by 
experts and items were selected for actual forms. 

h. Creation of Multiple Test Forms: Parallel forms were developed. 
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Appropriate cutoff scores have been established by the panels involved with 
the content review based on what was felt to be the minimum knowledge 
necessary to be a successful beginning teacher. However, states may 
establish their own cutoff scores. Cutoff scores for The Basic Skills Test has 
been determined to be 78.8 percent correct on the exam, and 75.3 percent on 
The Pedagogy Test. 

5. Technical and Psychometric Aspects 
Validity 

A great deal of time and effort was spent on insuring validity of the CMEE. 
The following orimary considerations that lOX used in developing the 
exam: 

* Goals: to measure appropriate content and to limit bias. 

* Five geographic regions: East Coast, South, Midwest, Pacific Northwest, 
and the Far West. 

* Defined content validity as "the degree to which the sample items, tasks, 
or questions on a test are representative of some defined universe or domain 
of content " (p. 26) as defined by Standards of Educational and Psychometric 
Testing (1985). 

* Focused on job-related questions. 

* NCS maintains that because of its high content validity, no local validity 
testing is really necessary. 

Content validity was established via opinions of educators and teacher 
education specialists. A statistical analysis was performed on each test to 
determine internal consistency, to establish content validity, and to 
determine bias using mean per item indices and point biserial coefficients 
for each section of each test. Each item of each section was rated by a panel 
of experts and the analysis was conducted on their ratings. Coefficients for 
The Basic Sldlls Test were .933 for content and .973 for bias-faimess on 
Form A and .939 for content and .977 for bias-faimess on Form B. 

Bias 

* The Bias Review Panels were composed of African American, Asian 
American and Hispanic American educators and assessment experts. 

* Questions asked of panelists: 

"Might this item offend or unfairly penalize any group of candidates 
on the basis of personal characteristics such as gender, ethnicity, religion, 
or socioeconomic status?" (p. 31, Development and National Validation 
Process). 
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"Considering the entire set of test items that you have just reviewed, 
do the items, taken as a whole offend or unfairly penalize any group of 
candidates on the basis of personal characteristics such as gender, 
ethnicity, religion or socioeconomic status?" (p. 33, Development and 
National Validation Process). 

• Native Americans were included in the bias review process, but not 
reported as part of the review sample. 

• NCS considers the CMEE to be bias free. 

Reiiabilitv 

Reliability is reported to be slightly different for each state, depending upon 
the state's cutoff scores. Coefficients were .92 for the pilot and .91 for all 
other testing. Total test reliability is around .92. 

Sflore Reporting 

Score reporting comes in two formu: The Individ ual Profile, which is sent 
directly to the candidate, and the State Sum mary Report which is sent to 
Institutions and State Departments for official recording of scores. 

Thfl Individual Profile has the following features: 

• Personal Information that the candidate can review for acf'uracy. 

• Total Test Results, giving the individual's score, the state cutoff score, 
and a pass/fail indicator. 

• Content Outline Results (for those taking the content area exams), 
showing the candidate specific content areas that require additional study. 

• For The Basic Skills Test an individual's performance levels are revealed 
relative to other candidates. These indices indicate whether scores fell 
below, the same as, or above the standard. Candidates can readily 
determine the areas that need further study in order to pass the exam. 

• Customized comments for each individual. 

The State Summary Report has three portions, including content cutline 
results for all candidates and two alphabetized rosters. 

• The Content Outline Renort shows an institution the number of 
candidates taking the exam, and the number and percentage of those 
passing and failing. 

• The Alphahetired Rosters include names of candidates, the test(s) taken, 
the scaled scores, and an indication of those failing. It also provides a 
separate report that outlines where each candidate needs additional study. 
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6. Strengths and Wealrjiesses 
Strenytha 

• Extensive Validity Studies 

• High reliability 

• Easily understood score reporting 

• Willingness of NCS to assist states to set up and administer tests. 
Wcftkncaacs 

• No data reporting how minority groups perform on the tests. 

Discussion on the CMEE 

CMEE is a valid and reliable criterion-referenced test that will indicate to 
an appropriate degree, which candidates possess the knowledge that expert 
opinion has indicated necessary for beginning teachers. NCS appears to be 
sensitive and responsive to state needs for teacher testing and appropriate 
outcomes. 

One of the goals of NCS regarding the CMEE is that it will serve to improve 
teacher education. By reviewing areas recommended for further study, 
institutions can review their students' performance in specific content. 

This can lead to improvement of instruction. This is an admirable goal, but 
may potentially lead colleges to "teach to the test". NCS may respond by 
saying that this may be appropriate, since CMEE was developed with the 
assistance of top experts in the field of education and teacher education. 

The question must be asked, do individual states and their teacher 
education programs set the standards or does a teacher evaluation 
instrument? 

In regard to NCS's claim that the CMEE is bias-free, the following should 
be considered: 

1. Regardless of bias review by panels of members of minority groups, the 
definition of bias is still subjective and it is a stretch to claim that any 
assessment instrument of this nature is bias-free. 

2. The Bias Review Panels did not include Native American 
representation, nor are there adequate data on Native American 
performance. 

To adequately address bias in Montana, a study should to be conducted with 
the Native American students in this state. 
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The Praxis Series 



1. General Description 

Prarig Sarieg’ Prnfeaaional Aflseasmenta for Beginning T eachers is 
intended to evaluate both teachers-in-preparation and beginning teachers 
in information and skills associated with teaching. The Praxis series 
consists of three assessment components and an instructional program. 
Pravta AffAfiftTnic Skills Assessments is intended to measure proficiency 
in the foundation skills of reading, writing and mathematics. These tests 
are designed to be taken diuing an individual’s college career, with the 
purpose of determining that prospective teachers have established basic 
enabling skills. Praxis 11: S ubject Assessments is designed to measure 
knowledge in specific teaching content areas. Praxis III: Classroom 
Parformanca Assessments aims to measure classroom performance of new 
teachers. The first two elements consist of paper and pencil exams (or a 
computer based alternative for Praxis I). The third is a structured 
observaticr of actual teaching conducted in conjunction with an interview 
of the teacher. T .earning Pius is ETS’s computer-based instructional 
program for adult learners who need help with basic reading, writing, and 
mathematics skills. The remainder of this report focuses on Prwas I and 
Prfl-Ha TT - the elements of the Praxis Series which address information 
related to initial entry into the teaching profession. 

Pr-axia T is offered in two formats. The Computer-Based Academic Skills 
Aaattaament requires a DOS system and can be used in single self-standing, 
multiple self-standing, or networked configurations. The computer-based 
format incorporates a range of question types including multiple response, 
highlighting, fill-ins, reordering, and candidate-constructed responses. 

The claim is made that the computer-based format tailors tests to smt 
individual candidate's performance insofar as “right” answers trigger 
more difficult questions and “wrong” answers trigger easier questions, 
with the purpose of accurately pinpointing skill levels. The computer-based 
exam can be taken at any time of the year and lends itself to individual or 
group administration. A more conventional paper-and-pencil format, 

.gjlrilla Teats is also available. In this exam, multiple-choice 
questions are based on reading passages and statements and having the 
examinee choose the most appropriate response among those provided. 

Both assessment formats also include the opportumty for candidates to 
construct an essay. The paper-and-pencil format is scheduled eight times a 
year and lends itself to group administration. 

LearringPlua is a tutorial tool for Praxis I . Through a series of diagnostic 
and p.acement tests, an individual candidate’s strengths and weaknesses 
in basic academic skill areas are assessed. Based upon the resmts of these 
assessments instruction in reading, writing, and/or mathematics can be 
structured and offered in a self-paced computer-based format. 



Prflxia TT is a paper-and-pencil assessment of content area information 
assumed to be necessary for the teaching of particular subject areas. These 
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assessments are reported to measure subject matter knowledge and 
knowledge about teaching that subject. Over 70 different exams are offered, 
ranging from Art Education to World Civilization. A feature of note is that 
there are 11 different content area tests related to the education of students 
with disabilities. Topics range from "Special Education" to "Special 
Education: Preschool/Early Childhood" and include tests specific to 
teaching individuals with particular t}rpes of disabilities. Each Praxis II 
Subject Assessment offers a multiple-choice core test and one or more 
candidate-constructed response modules. These exams are also offered up 
to eight times a year. 

The various elements in the Praxis Series are priced separately. There is a 
$20 registration fee which is assessed for taking one or more exams in a 
single day. Costs range from $40 for a single Praxis I preprofessional test 
in paper and pencil format to $105 for three tests in the Praxis I computer- 
based academic assessment. Individual Praxis II subject matter module 
assessments cost $55 for one-hour exams and $70 for two-hour long core 
exams. 

A configuration similar to the CMEE would be possible for a person 
certifying for Elementary and Special Education; 

• Praxis I $40.00 

• Praxis II: Elementary content (oue hour $55.00, two hour $70.00) 

Special Education content ($55.00 or $70.00) 

• Registration Fee $20.00 per test session 

• Since Praxis I and Praxis II are offered on two separate dates, 
minimum registration fees would be $40.00. 

• Total fees for this battery would be $190.00 to $220.00. 

2. Test Content 

Praxis I tests skills in reading, writing, and mathematics which are 
assumed to be required of all teachers, regardless of the subject matter or 
grade levels being taught. 

Praxis II purports to measure candidates’ knowledge of the subjects they 
will teach. More than 70 subjects are covered. In addition, ETS offers 
states the opportunity to customize subject exams through a 
modularization process. For example, it is possible to combine a two-hour 
multiple choice subject core with one or more optional modules which focus 
on particular aspects of the subject or on ways to teach the subject which 
rely on candidate-constructed responses. 

4 

3. Acconunodationa for Persons with Disabilities 

With its flexible scheduling and computer delivery approach, the computer- 
based format available for Praxis I potentially offers opportunities for 
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creative accommodations for persons with disabilities. Special 
arrangements for accommodations can be made for other Praria tests. 
Advanced notice of need for accommodations would be required to put those 
accommodations in place. Part of the burden for accommodations falls 
upon the test administrator or test administration center. 

4. Development 

In its development of the Praxis Series . ETS has followed technical 
standards for educational and psychological testing as established by the 
American Educational Research Association, the American Psychological 
Association and the National Council on Measurement and Evaluation. 
According to ETS literature, as part of the development efforts, El 3 
reviewed current methods used in teacher evaluation; developed and 
conducted job analyses of teaching to determine what knowledge, skills, 
and tasks are required at each stage of the beginning teacher’s career; and 
examined state teacher licensing requirements. Approximately 3,000 
teachers and other educational experts were consulted at various stages of 
the planning and development stages in the process in which they engaged. 
Many of these individuals were involved in initial job analyses in which 
opinions were gathered in regard to job-related knowledge and skills. 

To develop test questions, ETS did the following: 

a. Reviewed state licensing requirements 

b. Performed job analyses to determine important knowledge and 
skills for beginning teachers 

c. Test specifications were developed by panels of experts 

d. Test items were developed 

e. Items were validated by panels of experts 

f. The test was constructed 

g. Standards were set 

h. Tests were administered in several states 

i. Results were reviewed by states and institutions 
5. Technical and Psychometric Aspects 
Validity 

According to ETS, “these assessments are used as (emphasis is mine) 
of the qualifying process for state license and entrance into teacher 
education.” National validation is offered for Praxis I and multi-state 
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validation is offered for Praxis II (It has been pretested in 37 states and the 
Virgin Islands). Each state needs to establish validity when piloting the 
test. 

Bias 

It appears the efforts have been made throughout the development process 
to pay attention to the multicultural aspects of classrooms in which 
teachers teach. However, evidence is sparse in regard to specific efforts to 
attend to the issues of candidates from diverse cultural backgrounds who 
will be taking the Praxis exams. ETS did make use of a “Fairness Review 
Panel” to make judgments regarding the questions which were included in 
Praxis I . Acknowledgment is made that “judgments of fairness remain 
highly subjective.” The panel consisted of three individuals from Hispanic, 
Asian-American and African-American ethnic backgrounds. In addition, 
several of the assessments in Praxis II have been reviewed by additional 
consultants for fairness. 

Reliability 

• Reliability for Praxis I ranges between .85 - .89. 

• Reliability for Praxis II content areas vary. For example, based on an N 
of 11,000 the reliability estimate for elementary education (general) is .91 
with a standard error of measurement of 4.8. For Special Education 
(general) the reliability is .92 with a standard error of 5.1 based on an N of 
1,100. 

Score Reporting 

The computer-based academic skills assessment offers “unofficial” scores 
immediately upon completion of the exams for reading and mathematics. 
Official scores on these and other exams are mailed to candidates and 
specified score recipients within several weeks of test administration. Both 
individual scores and composite institutional scores are available. 

6. Strengths and Weaknesses 

Strengtha 

• The Praxia Series offers flexibility to individueil states, insofar as a 
“battery” could be specified that reflects a state’s specific licensing 
requirements. 

• A wide range of subject matter tests are available, including 11 in the 
education of students with disabilities. 

• The Liateniny Comprehension portion of the NTE is no longer included 
in the assessment process. 
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Weakneagea 

* Minority groups appear to be under-represented in validation studies. 

Of specific interest to the state of Montana is the fact that Native 
Americans are not reported as part of test development efforts. 

• Score reporting is not particularly user friendly for individuals who took 
the test; neither is it for teacher education institutions and state 
departments. 

Discuaaion of the Praxis Series 

The Prayia Series consists of valid and reliable tests constructed from job 
analyses that indicate, to an appropriate degree, which candidates possess 
the ^owledge that expert opinion has indicated necessary for beginning 
teachers. States can make choices as to which portions of the series are 
appropriate, and ETS claims they will assist states to establish a battery of 
tests to meet theii' assessment requirements. 

PraTia T covers basic skills in mathematics, reading and writing, while 
Prayia TT assesses 70 separate content areas. Praris III assesses teachers 
on the job. Careful guidelines and training for assessors are provided to 
assure fairness and consistency for those assessments. 

General Discussion 

There are numerous professions that require assessment of knowledge and 
skills prior to entrance and acceptance into that profession. They are part 
of society's expectation of the profession, as well as the profession itself. In 
some respects, the assessment procedure may be considered a privilege in 
which an individual is permitted to participate, or a rite of passage into the 
profession. Accountants, physicians, attorneys, counselors, and nurses 
are but a few of the numerous professions that require an assessment of 
skills and knowledge before allowing professional practice. These 
assessment tools serve to protect ».he public and the profession by 
determining basic and agreed upon domains of knowledge widely accepted 
by the professions that utilize them. It is therefore justifiable that teacher 
testing be an established point of entry to that profession as well. 

The CMEE and the PraTia Series are both appropriate assessment batteries. 
Both have been developed and studied to assure validity and reliability. Both 
relied on the Standards of Educationa l and Psychometric Testing during 
development and validation. However, a couple of words regarding 
technical assistance seem to be in order. During the development of this 
report, the National Computer Systems staff were readily available to 
answer questions, send information and explain the process of 
development, validation and procedures regarding the CMEE. On the other 
hand, ETS was more of a challeng j. It took two weeks to obtain reliability 
and validity coefficients, which should be readily available to inqmrers. 
Access to information on validity and reliability seemed to be dependent on 
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amount of biu-eaucratic-type transfer and run around before that individual 
was found. This inconvenience could be avoided if the information was 
included in the promotional materials and test manuals. The point is not to 
suggest that the decision to contract with one company over the other 
should be based on this individual's experience, but to emphasize that 
accessibly to information and technical assistance is an important issue to 
consumers, in general. 

An assessment of the knowledge of candidates for teac ling certificates 
must serve the state (representing the children) and the teaching 
profession by assisting to certify only qualified candidates. It should not, 
however, be a punishment for the candidates themselves. Confusing and 
cumbersome assessments and paperwork can be, at best, bothersome and, 
at worst, expensive and frustrating. Keeping the process simple and 
efficient, and assessing the areas the state and the profession deem 
necessary are important. Variables such as score reporting, cost, and 
accessibility ought to be considered when making decisions about two 
acceptable assessment instruments. 

The state must now make a decision on how to proceed. Here are a few 
guiding questions. 

• When is the most appropriate time to test for basic skills competency: 
early in one’s academic years or later? 

• How will minimum scores (floors) for each test be established, and 
which group(s) will make those determinations? 

• How and when is instructional competence assessed in relation to 
content knowledge? 

• Should professional associations play a more dominant role in the 
assessment process like they do in the certification and licensure of 
professionals in law, accounting, medicine, and others? 

• What role, if any, should state and national teacher education 
accrediting bodies play in the assessment process? 



